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Section  1 

Introduction 


Robot  control  systems  typically  consist  of  cooperating  micro-computers. 
Frequently  the  processors  use  shared  memory  on  a  common  bus  for  com- 
munication. This  is  particularly  true  when  the  processors  must  cooperate  in 
executing  low  level  tasks,  such  as,  servo  loops.  This  architecture  has  great 
appeal:  construction  of  the  controller  is  easy,  using  off-the-shelf  busses  and 
boards;  the  system  can  be  quite  modular  and  expandable  for  the  same  rea- 
son; shared  memory  is  conceptually  simple  and  flexible  for  the  programmer; 
and  shared  memory  is  a  reliable  and  fast  communication  medium.  Problems 
arise  as  robot  systems  get  more  complex.  There  are  mechanical  and  elec- 
trical limits  to  the  expansion  of  shared  memory  systems  which  put  rather 
small  bounds  on  the  number  of  boards  that  can  be  in  a  system.  Centralized 
processing  in  a  single  box  requires  extensive  cabling  to  the  various  sensors 
and  actuators.  This  is  often  awkward,  constraining  the  placement  of  a  robot 
and  the  inhibiting  expansion.  Complex  sensors,  now  in  development,  such 
as  tactile  sensing  arrays  compound  the  problem  because  of  the  multitude 
of  connections  required.  An  approach  to  the  problem  of  cabling  between 
sensors,  actuators  and  processors  is  to  put  the  processors  near  to  the  de- 
vices. This,  though,  poses  a  communication  problem  between  the  various 
processors,  among  the  processors  controlling  the  devices  and  between  these 
processors  and  higher-level  control  processors.  Ganglia  is  a  robot  controller 
architecture  and  communication  protocol  meant  to  address  these  issues.  Dis- 
tributing a  robot  controller  within  the  robot  presents  many  problems,  for 
instance,  those  related  to  weight,  size,  packaging,  power-distribution,  and 
software  tools.  The  current  research  on  ganglia  is  focused  on  the  commu- 
nications problems  and  only  lightly  touches  on  these  further  topics. 

The  impetus  for  ganglia  came  while  considering  the  design  of  a  con- 
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troller  for  the  Utah/MIT  Hand  [Joco84],  an  anthropomophic  hand  with 
four  fingers  one  of  which  is  opposed  to  the  other  three  and  acts  as  a  thumb. 
Each  finger  has  four  joints  for  a  total  of  16  degrees  of  freedom  and  each  joint 
has  a  position  sensor  and  a  torque  sensor.  The  pneumatic  cylinders,  two  per 
joint,  which  provide  power  for  the  joints  are  located  in  a  power  pack  located 
roughly  four  feet  from  the  hand  itself.  Tendons  connecting  the  joints  and 
the  cylinders  are  routed  through  a  flexible  linkage.  This  linkage  gives  the 
hand  the  freedom  to  move  about  in  a  cube  of  about  two  feet  per  edge  while 
the  power  pack  remains  stationary.  The  hand  is  controlled  by  a  controller 
consisting  of  several  micro-processors  on  a  common  bus,  an  architecture  very 
common  in  robot  controllers.  Sensory  data  runs  along  cables  connecting  the 
32  tension  sensors  and  16  position  sensors  in  the  hand  to  the  controller  and 
actuator  signals  are  sent  from  the  controller  to  the  32  cylinders  in  the  power 
pack.  The  hand  alone  is  a  complex  robot  by  contemporary  standards  but 
by  itself  it  is  of  very  limited  use.  To  provide  mobility  the  hand  must  be 
attached  to  an  arm  which  adds  several  more  degrees  of  freedom  and  sensing 
to  the  system.  To  use  the  hand  for  dextrous  manipulation  tactile  sensor 
must  be  added  to  finger  tips  [J0C088]  [SpeeSS].  These  add  a  great  deal  of 
complexity  to  the  system.  This  system,  the  Utah/MIT  hand,  an  arm,  and 
tactile  sensors  is  quite  complex  but  it  would  be  short  sighted  to  think  this  is 
in  any  sense  the  ultimate  in  robot  complexity,  even  if  only  the  near  future  is 
considered.  One  of  the  many  problems  that  arise  when  considering  complex 
systems  is  how  the  various  components  are  connected.  The  typical  robot 
controller  today  is  a  single  box  or  rack  containing  multiple  micro-processors 
and  each  component  has  its  own  connection  to  the  controller  or  perhaps  a 
few  sensors  share  a  single  connection.  To  further  complicate  matters,  it  is 
in  the  nature  of  robot  arms  that  the  most  interesting,  delicate,  and  complex 
component  is  usually  the  end-effector,  wliich  is  farthest  from  the  centred 
controller.  Thus  the  most  numerous  and  most  delicate  signals  must  travel 
the  greatest  distance  subjecting  them  to  more  interference  both  electrical 
and  mechanical.  An  obvious  approach  to  this  problem  is  to  distribute  the 
controller  within  the  robot,  to  put  the  processors  near  to  the  sensors  and 
actuators.  This  poses  a  new  problem:  How  do  these  processors  communi- 
cate with  each  other  and  with  higher  level  control  processors?  This  is  the 
problem  addressed  by  ganglia. 

The  project's  vision  is  that  one  might  someday  have  components  the  size 
of  a  matchbook  capable  of  the  low  level  control  of  actuators  and/or  sensors. 
This  would  entail  the  necessary  interface  electronics,  processing  power,  and 
communication  electronics.  A  single  cable  would  connect  the  component  to 
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the  network  of  components  and  provide  power  for  the  component.  These 
components  or  nodes  would  be  scattered  about  the  robot  as  needed.  Other 
nodes,  more  centrally  located,  would  provide  coordination  and  higher  level 
control  and  communicate  with  still  higher  controllers  (eg.  the  operator,  a 
multi-robot  controller).  Nodes  are  not  restricted  to  these  matchbook  size 
components,  high  level  control  or  communication  nodes  or  notes  requiring 
intense  computation  may  be  full-blown  computers.  Each  "independent" 
robot,  for  instance  an  arm  and  its  end-effectors,  would  have  its  own  network 
with  gateway  nodes  to  other  robots  if  close  coordination  is  required.  On 
complex  systems  one  might  expect  dozens  of  nodes  while  in  some  instances 
hundreds  may  be  required.  Much  needs  to  be  done  before  such  a  vision  is 
realized.  Developing  a  communication  architecture  suitable  for  this  network 
is  an  important  step  in  this  direction. 

Section  2  presents  a  overview  of  the  problems  presented  by  real-time  com- 
munications and  a  brief  survey  of  the  nature  of  inter-process  communication 
in  current  robot  control  systems.  The  next  section  introduces  a  t«ixonomy  of 
the  types  of  messages  expected  in  a  distributed  robot  controller.  The  next 
three  sections  (4  through  6)  describe  in  some  detail  the  major  components 
of  the  GANGLIA  system,  the  network  itself,  the  communication  protocol,  and 
the  communication  memory  management  unit.  Section  7  presents  brief  dis- 
cussions of  alternatives  for  various  parts  of  the  GANGLIA  system  and  some 
possible  extensions.  Finally,  Section  8  is  the  conclusion. 


Section  2 

Real-time  Communication 


The  purpose  of  GANGLIA  is  to  support  robot  control  programs.  This  section 
presents  briefly  the  important  characteristics  of  robot  control  programs.  For 
a  more  detailed  survey  of  operating  systems  for  robot  controllers  and  control 
program  characteristics  see  Operating  Systems  for  Robot  Control  [Clar88]. 


2.1      Constraints 

The  most  prominent  characteristic  of  robot  control  programs  is  that  they 
must  meet  rejil-time  constraints.  Robots  often  operate  in  restricted  environ- 
ments but  the  rea/ world  can  not  be  kept  entirely  at  bay.  Gravity,  unexpected 
obstacles,  mechanical  imprecision,  and  mechanical  failure  are  features  of  the 
environment  that  a  robot  and  its  controller  must  cope  with.  A  robot  grasp- 
ing a  delicate  object  must  continuously  monitor  the  forces  it  applies  to  the 
object  to  avoid  damaging  the  object  with  too  much  force  or  dropping  the 
object  by  not  applying  enough  force.  In  this  situation  the  controller  must 
detect  excessive  or  insufficient  force  and  respond  before  the  object  is  dam- 
aged or  gravity  pidls  the  object  away.  It  is  of  course  sometimes  possible  to 
develop  a  special  gripper  that  by  its  design  prevent  the  object  from  slipping 
or  being  damaged.  However,  this  limits  the  generality  of  the  robot  gripper 
and  thus  the  tasks  that  can  be  performed.  Real-time  constraints  also  arise 
in  the  normal  operation  of  robots  not  just  under  exceptional  conditions  such 
as  a  slipping  object.  Moving  a  robot  arm  along  a  smooth  path  to  a  specific 
point  requires  frequent  comparison  of  actual  joint  positions  to  target  posi- 
tions and  if  the  motion  is  to  be  smooth  the  controller  must  respond  in  a 
timely  manner  to  the  deviations.  A  robot  controller  must  be  able  to  operate 
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within  both  kinds  of  time  constraints  to  be  effective. 

A  network,  such  as  GANGLIA,  which  intends  to  support  real-time  pro- 
gramming in  turn  has  these  real-time  constraints  imposed  on  it.  In  fact,  the 
success  of  a  real-time  protocol  can  be  measured  as  the  percentage  of  mes- 
sages that  are  delivered  on  time  (see  Multiple- Access  Protocols  and  Time- 
constrained  Communication  [KuRo84]).  Performance  evaluation  criteria  for 
normal  packet  switching  networks  are  the  throughput  or  messages  delivered 
per  unit  time  and  the  trade-off  between  message  throughput  and  the  aver- 
age time  delay  between  a  message's  generation  and  reception.  In  a  real-time 
network  though  a  message  that  arrives  too  late  is  often  of  no  value  so  the 
delay/throughput  trade-off  is  a  dubious  measure.  GANGLIA  has  determin- 
istic scheduling  for  the  most  part  and  intends  to  deliver  every  message  on 
schedule.  There  are  two  instances  where  this  is  not  possible:  One  is  when 
a  message  is  corrupted  electrically  in  transmission  and  the  other  is  is  when 
the  load  exceeds  the  capacity  of  the  network  and  protocol.  In  the  first  case 
the  corrupted  message  is  lost  and  there  is  no  attempt  on  ganglia's  part 
to  re-transmit  the  message.  The  assumption  is  that  this  sort  of  corruption 
is  infrequent  so  that  attempts  to  overcome  it,  at  this  level  in  the  network, 
would  be  too  costly.  Attempts  to  ensure  the  delivery  of  messages  requires 
that  any  message  may  have  to  be  transmitted  an  arbitrary  number  of  times. 
This  though  introduces  uncertainty  into  the  schedule  which  goes  against 
the  desire  for  deterministic  or  predictable  scheduling.  In  addition,  much  of 
the  traffic  in  a  robot  controller  is  not  overly  sensitive  to  lost  messages,  this 
is  discussed  a  bit  more  below,  so  it  is  appropriate  for  ganglia  to  provide 
an  unreliable  datagram  service  instead  of  a  guaranteed  service.  When  the 
load  exceeds  the  capacity  during  critical  periods  ganglia  filters  out  the  less 
important  messages  using  a  priority  scheme  similar  to  hardware  interrupt 
levels  used  in  most  processors.  Furthermore,  a  possible  extension  to  GAN- 
GLIA presented  in  Section  7  is  a  distributed  protocol  for  detecting  network 
congestion  and  filtering  out  less  important  traffic.  There  are  two  reasons 
why  deterministic  scheduling  is  feasible  within  a  robot  controller,  one  is 
that  it  is  a  closed  system  in  that  the  network  itself  is  not  dynamically  modi- 
fied and  the  other  is  that  much  of  the  activity  is  very  regular  and  predictable 
(eg.  servo  loops).  Thus  for  a  given  system  a  suitable  schedule  for  the  net- 
work traffic  can  be  computed  off-line  and  compiled  into  the  system.  Of 
course,  unexpected  emergencies  or  exceptional  conditions  can  not  be  deter- 
ministically  planned  for,  instead  GANGLIA  is  careful  about  when  exceptional 
conditions  can  be  raised  within  the  system  and  then  immediately  adjusts 
the  network  traffic  as  needed  to  best  respond  to  the  exception.  This  can  be 
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likened  to  a  typical  processors  response  to  hardware  interrupts.  Interrupts 
are  only  permitted  between  instructions  when  the  processor  is  in  a  "simple" 
and  restorable  state  and  then  the  interrupt  is  handled  by  switching  to  a 
pre-planned  routine  which  is  to  appease  the  interrupting  device  and  return 
to  the  interrupted  program  flow  as  soon  as  is  possible. 


2.2     Inter-process  Communiction. 

An  important  discriminating  characteristic  of  robot  control  systems  is  the 
types  of  inter-process  communication  available  or  employed.  The  typical 
architecture  for  robot  controllers  is  multiple  computers  or  processors  on  a 
common  bus.  The  major  components,  processors  and  devices,  are  connected 
to  the  bus  and  most  communication  is  via  shared  memory  although  other 
communication  formalisms  are  often  placed  over  the  shared  memory.  Fre- 
quently, though  not  universally,  multiple  tasks  run  on  each  of  the  processors 
and  these  tasks  control  the  robot.  For  the  most  part  the  tasks  are  statically 
allocated  to  the  processors  and  do  not  migrate  among  the  processors  and 
often  they  are  neither  killed  nor  spawned  but  are  created  at  compile  time 
and  exist  for  the  duration  of  the  system.  This  static  nature  of  processes  or 
tasks  is  a  reflection  of  the  closed  nature  of  robot  control  systems  and  can 
simplify  inter-process  communication. 

'Messages  passed  on  queues'  a  common  inter-process  communication  par- 
adigm and  is  used  in  some  robot  control  systems.  Harmony  [Gent84]  uses  it 
extensively  while  NRTX  [Kapi84]  and  SAGE  [Salk88]  support  it.  Harmony 
constructs  an  entire  real-time  programming  style  around  message  passing 
[Gent81].  Great  care  is  taken  so  that  processes  do  not  block  unexpectedly. 
For  instance  the  sending  process  (i.e.  the  process  initiating  an  exchange) 
supplies  all  the  buffer  area  needed  for  the  complete  message  exchange  so 
that  no  process  need  access  a  buffer  or  message  pool  since  an  empty  pool 
would  require  that  the  process  be  blocked.  The  sending  process  blocks  until 
the  processing  of  its  message  is  completed  by  a  reply  from  the  destination 
process.  A  receiving  process  blocks  only  if  there  are  no  messages  for  it  to 
handle.  Eliminating  unexpected  blocks  removes  much  uncertainty  about 
how  long  it  will  take  to  process  a  request  for  a  task  so  that  the  programmer 
can  construct  the  task  to  meet  critical  deadlines.  However,  queues  usually 
mean  some  uncertainty  in  time  since  there  may  be  several  messages  ahead 
of  a  tasks  message.  In  the  examples  of  Harmony  [GentSI]  programs,  this  is 
avoided  by  programming  in  a  way  which  ensures  that  critical  queues  never 
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contain  more  than  one  message.  The  message  passing  paradigm  is  largely 
independent  of  the  communication  medium  and  it  is  easily  implemented  on 
shared  memory  systems  by  simply  passing  pointers  to  the  actual  message. 

A  common  form  of  inter-process  communication  is  simply  signaling  an 
event  with  little  if  any  additional  data.  An  example  from  a  time  sharing 
system  of  this  type  of  communication  is  signals  in  UNIX  [Kern  84]  which  are 
conceptually  similar  to  hardware  interrupts.  NRTX  which  is  a  descendent 
of  UNIX  provides  such  signals.  SAGE  and  Condor  [Sieg85][Sieg86],  a 
low  level  operating  system  for  the  Utah/MIT  hand,  provide  signaling  via 
mailbox  interrupts  by  which  one  processor  can  cause  an  interrupt  on  another 
processor  (or  to  itself)  and  transmit  a  single  word  of  data  in  the  process.  The 
appeal  of  signals  is  that  for  many  types  of  events  asynchronous  signaling  that 
the  event  has  occurred  is  all  that  is  needed  and  the  efficient  implementation 
of  signals  can  simplify  the  programming  around  such  events.  An  example 
of  such  an  event  is  the  updating  of  target  values  for  a  servo  process.  The 
generation  of  a  new  targets  may  well  be  infrequent  and  asynchronous  with 
respect  to  the  servo  process's  loop  so  that  checking  for  a  new  value  within 
the  loop  itself  may  be  inefficient  and  awkward.  An  asynchronous  signal  that 
the  new  target  value  is  available  is  an  efficient  and  elegant  alternative.  The 
most  severe  limitation  of  signals  is  that,  at  least  in  their  pure  form,  only  one 
bit  of  information  is  transmitted. 

Process  synchronization  is  a  crucial  form  of  inter-process  communica- 
tion. How  synchronization  is  presented  to  the  programmer  is  another  dis- 
tinguishing characteristic  of  real-time  operating  systems.  In  Harmony  syn- 
chronization is  implicit  within  the  message  passing  since  the  sending  of  a 
message  is  not  complete  until  the  receiver  acknowledges  it.  In  NRTX  and 
SAGE  synchronization  is  available  in  several  forms  including  messages,  sig- 
nals, test-and-set  instructions  and  semaphores.  Condor  does  not  provide 
synchronization  directly  but  the  synchronization  facilities  of  the  underlying 
hardware  are  available,  i.e.  the  test-and-set  instruction.  In  fact,  process 
blocking  is  virtually  impossible  in  Condor  because  a  single  stack  is  used  for 
all  processes  on  a  processor.  NYMPH  [Chen 86]  another  low  level  operating 
system  provides  facilities  for  multiple  processors  to  participate  in  synchro- 
nization. In  fact,  this  is  about  the  only  service  NYMPH  provides.  Modern 
micro-processors  and  busses  support  inter-processor  synchronization  with 
a  test-and-set  or  similar  operation  that  eliminates  the  ambiguities  possible 
with  simultaneous  references  to  a  common  variable  by  multiple  processors. 

As  mentioned  previously,  the  typical  robot  control  system  is  implemented 
on  multiple  micro-processors  on  a  common  bus  and  in  these  systems  shared 
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memory  is  the  underlying  communication  medium.  For  all  the  mentioned 
systems  except  Harmony  shared  memory  is  the  primary  method  of  passing 
all  but  small  amounts  of  data  and  even  in  Harmony  shared  memory  facili- 
tates the  implementation  of  its  message  passing  primatives.  This  architec- 
ture has  great  appeal:  construction  of  the  controller  is  easy,  using  standard 
busses  and  boards;  the  system  can  be  quite  modular  and  expandable  for  the 
same  reason;  shared  memory  is  a  reliable  and  fast  communication  medium; 
it  is  conceptually  simple  and  flexible  for  the  programmer;  and  other  com- 
munication formalism  can  be  implemented  on  top  of  shared  memory. 

Ganglia  is  a  suitable  communication  base  for  supporting  all  of  these 
forms  of  communication,  including,  in  part,  shared  memory.  It  should  be 
noted  that  the  ganglia  architecture  does  not  specify  what  operating  system 
is  running  on  the  individual  nodes  and  a  ganglia  system  is  likely  to  have  a 
heterogeneous  collection  of  operating  systems  on  the  nodes.  Consequently, 
ganglia  trys  to  support  a  very  broad  range  of  communication  styles  in 
an  efficient  manner.  To  this  end  ganglia  implements  a  network  or  global 
memory.  There  is  an  address  space  common  to  all  the  nodes  on  the  network. 
A  node  writes  to  this  address  space  by  putting  the  data  on  the  network  along 
with  the  address  of  the  data  and  all  the  nodes  then  have  access  to  the  data. 
The  memory  is  not  implemented  as  a  separate  entity  on  the  network  but  each 
node  maintains  the  portion  of  the  memory  in  which  it  is  interested.  Parts 
of  the  memory  are  thus  duplicated  in  various  nodes  of  the  network.  This 
implements  an  approximation  to  a  shared  memory  on  the  network  which 
is  deficient  in  two  main  regards,  when  compared  to  memory  on  a  common 
bus.  First,  the  delay  in  communication  which  means  that  changes  to  the 
memory  are  not  "instantaneous"  and  thus  a  cause  for  concern  in  time  critical 
processes.  Second,  is  because  the  memory  is  duplicated  on  the  various  nodes 
along  with  unavoidable  communication  failure  and  delay  the  global  memory 
may  become  inconsistent,  that  is  different  nodes  seeing  different  values  in 
the  memory.  Many  of  the  details  of  the  GANGLIA  design  are  intended  to 
contain  or  minimize  these  deficiencies.  To  facilitate  inter-process  and  inter- 
processor  signaling  the  processor  in  a  node  may  request  to  be  interrupted 
when  a  particular  item  of  data  appears  on  the  network  (i.e.  when  the  datum 
is  changed).  And  to  provide  synchronization  between  processors  GANGLIA 
implements  at  a  low  level  a  message  acknowledgement  scheme  and  a  test- 
and-set  message.  Again,  because  of  potential  communication  failure  these 
are  not  as  reliable  as  similar  facilities  between  processors  on  a  common  bus 
but  with  a  reliable  communication  medium  the  ganglia  implementations 
should  provide  useful  and  efficient  methods  for  process  synchronization. 


2.2.    Inter-process  Communiction. 


Two  additional  characteristics  of  robot  control  programs  affect  the  com- 
munication. Both  are  artifacts  of  servo  loops  which  are  basic  to  the  low-level 
control  of  robot  systems.  First  is  the  periodicity  of  servo  loop  messages. 
Servo  loops  run  within  strict  intervals  and  the  communication  traffic  associ- 
ated with  them  will  follow  similar  patterns.  The  communication  scheduler 
can  take  advantage  of  this  since  the  period,  timing,  and  communication 
traffic  for  the  servo  loops  is  known  in  advance.  The  second  characteristic 
is  that  much  of  the  servo  loop  traffic  is  repeated  reporting  of  continuously 
varying  values  such  as  joint  position  or  force  sensed  by  an  end  effector.  The 
importance  of  this  for  the  communication  system  is  that  an  occasional  lost 
message  is  not  likely  to  be  disastrous.  This  assertion  is  reasonable  since 
continuous  values  are  not  likely  to  change  much  in  one  cycle  so  that  the 
previous  value  can  be  used  instead  of  the  lost  value  and  since  an  updated 
value  will  be  received  in  the  next  cycle  the  servo  can  recover  during  the  next 
cycle.  The  consequence  is  the  many  servo  messages,  very  likely  the  most 
frequent  type  of  message  on  the  network,  do  not  require  acknowledgement, 
assuming  that  the  underlying  communication  is  reliable. 


Section  3 


Message  Taxonomy 


In  order  to  characterize  the  communication  traffic  formally,  the  following 
taxonomy  of  packets  or  messages  is  proposed.  First  messages  fall  into  one 
of  two  categories  based  on  their  regularity. 

Periodic  messages.  These  happen  at  regular  known  intervals.  This  class 
is  important  because  the  communication  network  can  take  ad- 
vantage of  their  periodicity  to  improve  performance.  Periodic- 
ity is  most  important  when  the  frequency  is  high  so  that  low 
frequency  periodic  messages  may  sometimes  be  considered  as 
sporadic  messages. 

Sporadic  messages.  From  the  standpoint  of  the  network  these  messages  oc- 
cur at  random.  There  is  very  little  the  network  can  do  to  pre- 
pare for  these  messages  except  perhaps  on  a  statistical  basis. 

Messages  fall  into  four  categories  based  on  their  relation  to  the  state  of 
the  system. 

Servo  messages.  Typical  of  these  messages  are  raw  sensor  values  or  actuator 
commands.  They  are  high  frequency  periodic  messages  that 
are  critical  to  the  system's  integrity  and  as  such  make  up  a 
background  traffic  that  must  always  run.  If  this  traffic  stops 
then  control  of  the  robot  in  any  real  sense  has  been  lost.  In 
current  robot  systems  actuators  and  sensors  are  usually  tightly 
coupled  to  the  processor  that  controls  them  at  this  level  so  that 
loss  of  control  at  this  level  is  uidikely.  In  a  GANGLIA  system, 
however,  it  might  well  be  that  actuators  and  their  controlling 
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processors  communicate  over  the  network  so  that  disruption  of 
the  net  may  disable  the  most  basic  control. 

A  variant  of  servo  messages  are  instrumentation  messages,  mes- 
sages used  to  record  certain  values  for  later  analysis  or  for  op- 
erator feedback.  These  messages  may  be  periodic  and  of  high 
frequency  like  the  servo  messages  but  they  are  not  as  critical  to 
the  system  integrity. 

Steady  state  messages.  These  messages  are  likely  to  be  periodic  but  at  a 
lower  frequency  than  the  above  messages.  They  help  maintain 
some  global  state  of  the  system.  Consider  an  arm  grasping  an 
object  and  following  a  specified  trajectory  in  free  space.  One 
would  expect  periodic  messages  that  ensure  that  the  grasp  is 
properly  maintained  and  that  the  progress  along  the  path  is 
appropriate  or  indicate  that  the  path  shouM  be  changed.  These 
are  steady  state  messages. 

Changing  state  messages.  These  messages  indicate  or  cause  significant  mod- 
ifications in  the  system  state.  Their  source  is  a  point  higher  in 
the  control  hierarchy  and  are  typically  sporadic.  Their  signifi- 
cance is  that  the  communication  network  may  be  able  to  change 
its  characteristics  to  better  suit  the  changing  state.  Using  the 
above  example,  when  the  object  contacts  a  surface  which  is  not 
unexpected  then  one  could  predict  a  flurry  of  communication 
activity  while  the  robot  system  changes  from  the  free  motion 
state  to  the  constrained  motion  state.  The  interim  state  may 
be  relatively  long  in  duration  while  parameters  are  calculated 
and  loaded  and  processes  are  started  and  stopped.  During  this 
period  the  communication  traffic  may  be  considerably  different 
than  during  the  free  motion  state  and  the  traffic  pattern  for  the 
constrained  motion  state  may  be  different  still. 

Exception  messages.  These  are  messages  similar  to  hardware  interrupts  or 
UNIX  signals  in  their  effect  on  the  system  state.  The  messages 
must  have  high  priority.  They  are  likely  to  precipitate  a  lot 
of  network  traffic  and  require  special  attention.  As  mentioned 
above  ganglia's  approach  is  to  field  these  messages  at  select 
times  that  minimize  disruption  to  the  network  schedule  and 
then  to  aid  in  the  resolution  of  the  exception  and  the  return  to 
"normal"  operation. 


12 Message  Taxonomy 

3.1      Frequency  and  priority. 

In  the  above  categories  one  would  expect  that  the  frequency  of  messages 
would  be  as  follows,  in  decreasing  order: 

•  Servo 

•  Steady  state 

•  Changing  state 

•  Exception 

With  servo  messages  being  the  most  frequent  and  exceptions  being  very 
rare,  hopefully. 

Priority  on  the  other  hand  should  be  assigned  as  follows,  in  decreasing 
order: 

•  Servo 

•  Exception 

•  Steady  state 

•  Changing  state 

The  relative  priority  of  steady  state  and  changing  state  messages  is  arguable 
but  the  significance  of  ranking  the  servo  messages  higher  than  the  exceptions 
is  that,  as  mentioned  above,  disruption  of  the  servo  messages  may  mean  total 
loss  of  control  which  would  complicate  or  prevent  an  appropriate  response 
to  the  exceptional  condition.  An  assumption  here  is  that  the  servo  traffic 
is  not  so  dense  that  an  exception  cannot  be  raised  until  it  is  too  late.  If 
such  a  situation  arises  the  network  is  clearly  overburdened  and  a  significant 
restructuring  of  the  network  and  control  system  is  necessary. 


Section  4 


The  Network 


This  and  the  following  two  sections  describe  more  concretely  the  design  of 
the  GANGLIA  network,  the  protocol,  and  the  Communication  Memory  Man- 
agement Unit.  It  is  useful  to  keep  in  mind  examples  of  GANGLIA  systems. 
A  simple  system  could  consist  of  a  six  degree  of  freedom  robot  arm  with  a 
complex  end  effector,  such  as  the  Utah/MIT  hand  [Joco84].  It  is  impor- 
tant to  remember  that  the  controller  for  the  robot  is  distributed  around  the 
robot  so  that  the  processor  controlling  the  actuator  for  a  joint  may  be  in  a 
different  node  than  the  position  sensor  for  the  same  joint.  The  end  effector 
need  not  be  a  manipulator  but  could  be  a  sensor,  for  instance  an  image 
processing  system,  and  again  at  least  some  of  the  processing  of  the  sensory 
data  is  done  near  the  sensor.  Also  the  problem  is  not  very  interesting  if  the 
arm  is  simply  a  positioning  device  for  the  end  effector  so  it  is  assumed  that 
there  is  real-time  interaction  between  the  end  effector,  its  task,  and  the  arm. 


4.1      The  Physical  Layer. 

The  physical  layer  for  GANGLIA  is  not  yet  specified  but  some  characteristics 
can  be  stated.  The  medium  is  to  be  a  high  speed  serial  bus.  The  use  of 
a  parallel  bus,  say  eight  bits  wide,  has  been  considered  but  at  present  the 
assumption  is  that  a  serial  bus  is  used.  The  bandwidth  of  the  network  is  yet 
to  be  determined.  Most  certainly  any  initial  experiments  will  be  performed 
on  twisted  pair  or  coaxial  cables  with  data  rates  of  around  10  Megabits. 
Preliminary  calculations  indicate  that  this  should  be  sufficient  for  networks 
of  a  few  dozen  nodes.  To  support  networks  with  hundreds  of  nodes  a  different 
technology,  perhaps  fiber-optics,  will  be  necessary.   The  intention  is  that  a 
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GANGLIA  network  be  restricted  to  a  "single"  robot,  thus  that  the  network 
does  not  need  great  length.  A  maximum  length  of  100  meters  is  considered 
to  be  sufficient.  Unlike  most  local  area  networks  the  cable  for  a  GANGLIA 
network  is  likely  to  be  subject  to  frequent  flexing,  for  instance,  consider  the 
section  of  a  cable  passing  an  arm's  elbow.  Selecting  a  cable  able  to  withstand 
this  abuse  may  over-shadow  all  other  constraints  on  the  physical  layer. 

4.2  Data  Link  Layer. 

The  data  link  layer  has  also  not  been  considered  in  detail  yet.  The  re- 
quirements are  good  reliability  with  low  overhead.  Messages  in  a  GANGLIA 
network  will  be  small  so  that  any  per  message  overhead  has  a  large  effect  on 
the  network's  effective  throughput.  Detection  of  transmission  errors  is  im- 
portant so  that  some  form  of  redundancy  check  is  necessary.  Finally,  there  is 
the  question  of  collision  detection.  Currently  it  is  assumed  that  a  transmit- 
ting node  can  detect  a  collision  with  another  message  and  then  terminate 
transmission.  It  is  very  possible  that  this  is  not  needed  at  all  and  the  ability 
to  detect  collisions  can  be  removed  if  other  considerations  indicate  it  to  be 
too  expensive. 

4.3  Node  Architecture. 

A  block  diagram  of  a  ganglia  node  is  shown  in  Figure  4.1.  There  are 
five  major  components,  the  device  interface,  the  processor,  the  processor's 
memory,  the  communication  memory,  and  the  communication  memory  man- 
agement unit.  The  device  interface  is  the  nodes  connection  to  the  device  or 
devices  it  controls,  most  likely  this  is  some  D/A  or  A/D  converters.  Some 
nodes  are  computational  nodes  and  may  not  be  connected  to  any  device. 
The  processor  controls  the  node.  The  memory  is  simply  the  processor's  lo- 
cal memory.  The  communication  memory  is  where  messages  are  stored  into 
and  transmitted  from.  It  is,  in  fact,  the  portion  of  the  global  memory  (see 
Section  2)  maintained  locally.  Finally,  the  communication  memory  man- 
agement unit  (CMMU)  maintains  the  communication  memory  and  is  the 
node-level  interface  to  the  network.  The  CMMU  is  described  in  detail  in 
Section  6. 


4.3.    Node  Architecture. 
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Figure  4.1:  Node  Architecture 


Section  5 


The  Protocol 


For  discussion  of  the  network  protocol  a  simple  abstract  example  system 
will  be  used.  It  is  pictured  in  Figure  5.1.  Node  C  is  the  control  node  which 
will  be  described  later.  Node  A  is  a  higher  level  node  or  perhaps  a  link 
to  a  host  computer  running  the  high  level  control  program.  The  nodes  X, 
y,  and  Z  are  sensory  nodes  consisting  of  a  processor  connected  to  some 
Analog-to-Digital  converts.  The  remaining  nodes,  M  and  A'^,  are  actuator 
nodes.  As  with  the  sensory  nodes  they  contain  a  processor  and  the  necessary 
electronics  to  drive  an  actuator. 


5.1      Messages. 

This  section  describes  the  structure  of  GANGLIA  messages.  Table  5.1  de- 
scribes the  fields  of  a  GANGLIA  message  excluding  any  header  or  trailer 
information  for  the  data  link  frame. 


5.1.1      Named  messages. 

Messages  in  ganglia  are  not  addressed  to  particular  destinations.  Instead 
each  message  has  in  its  header  the  name  of  the  data  in  the  message,  these 
are  called  named  messages.  In  the  example  (Figure  5.1)  consider  the  node 
X  which  monitors  and  reports  some  sensors,  say,  for  joint  positions  for  the 
joints  m  and  n.  Periodically  this  node  must  transmit  a  message  or  mes- 
sages containing  the  current  positions  to  nodes  M  and  N  which  calculate 
corrections  to  the  position  actuators  based  on  these  sensed  positions  and 
then  adjust  the  joints.  On  a  typical  broadcast  network  the  straightforward 
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Field  Name 

Bytes 

Description 

Next 

2 

Node  identifier  of  the  next  node  to  transmit. 
This  and  the  Thread  make  up  the  token  (see 
Section  5.3). 

Thread 

1 

A  modifier  for  the  Next  node  (see  section  5.3). 

Flags 

1 

Flags: 

Empty  Message 

Exception 

Acknowledgement  Requested 

Message  Type:  (5  bits) 

Exception  Level  0-7 

Overrun 

Exception  Poll 

Idle  Network 

Clear  Test  and  Set 

Test  and  Set 

Test  and  Set  Reply  (0) 

Test  and  Set  Reply  (1) 

Positive  Acknowledgement 

Negative  Acknowledgement 

Name 

3 

Name  for  the  data  in  the  message,  this  name 
is  unique  across  the  network. 

Length 

1 

The  number  of  bytes  of  data,  a  value  of  zero 
indicates  256  bytes  of  data,  if  the  data  field  is 
empty  the  Empty  Message  flag  is  set. 

Source 

2 

Node  identifier  of  the  node  transmitting  the 
message. 

Cycle 

2 

A  time  stamp  on  the  data.   Each  time  a  par- 
ticular data  item  is  transmitted  the  Cycle  is 
incremented. 

Data 

0-256 

The  data. 

Table  5.1:  Message  Structure 


5.1.    Messages. 19 


approach  would  be  to  transmit  two  messages  with  the  address  of  the  desti- 
nation node  (M  in  one,  N  in  the  other),  some  indication  of  what  data  is  in 
the  message  (typically  a  communication  port  or  process  number),  and  the 
actucil  data  (the  positions  of  joints  m  and  n).  In  a  network  with  multicast 
capabilities,  a  single  message  could  be  transmitted  containing  the  multicast 
address,  some  indication  of  the  data,  and  the  data  itself.  Both  M  and  N 
will  receive  the  single  message.  In  ganglia  a  single  message  is  transmitted 
containing  the  name  of  the  data  ("Joint  positions  for  m  and  n")  and  the 
data  itself.  Every  node  on  the  network  receives  the  message  and  decides 
whether  or  not  it  is  interested  in  the  joint  positions,  if  so  the  value  is  stored 
otherwise  the  data  portion  of  the  message  is  discarded.  Clearly,  if  the  pro- 
cessor in  each  node  must  examine  every  message  and  decide  if  it  is  of  interest 
then  a  great  deal  of  processing  power  will  be  wasted  on  the  communication. 
The  CMMU  in  each  node  removes  this  burden  from  the  processors. 

In  Table  5.1  the  data  "name"  is  contained  in  foui  fields.  Name  is  the 
data's  name;  essentially,  it  is  the  location  of  the  datum  in  the  global  memory. 
Length  is  the  length  of  the  data  item,  the  maximum  is  256  bytes.  This  also 
corresponds  to  the  page  size  in  the  globed  memory.  Source  is  the  identifier 
of  the  source  node.  It  is  included  as  part  of  the  name  so  that  if  a  data  item 
has  more  than  one  source  the  receiving  nodes  can  determine  the  source. 
Cycle  is  a  time  stamp,  each  time  a  node  transmits  an  item  it  increments 
the  counter  for  that  item.  This  can  be  used  to  detect  inconsistencies  in  the 
global  memory  and  to  detect  data  that  has  gone  stale. 

5.1.2  Addressed  messages. 

It  should  be  noted  that  the  named  message  scheme  can  support  normal  "ad- 
dressed" message.  All  that  is  necessary  is  to  assign  one  page  of  the  global 
memory  to  each  of  the  nodes.  Then  messages  indented  for  a  particular  node 
are  sent  to  the  page  assigned  for  that  node.  Of  course,  each  node  must 
be  prepared  to  receive  data  for  its  page.  The  significance  of  simulating  ad- 
dressed messages  is  that  the  named  messages  scheme  does  not  preclude  any 
acknowledgement  or  synchronization  protocols  available  in  typical  networks. 

5.1.3  The  token. 

The  first  two  fields.  Next  and  Thread,  make  up  the  token.  Each  message 
contains  a  token  that  indicates  which  node  is  to  transmit  next  and  a  hint 
as  to  what  should  be  transmitted.  Thus  each  message  grants  access  to  the 
network  to  the  node  in  the  token.  A  node  receiving  the  token  is  obligated  to 
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transmit  a  message  immediately,  otherwise  it  will  be  assumed  that  the  token 
has  been  lost  and  corrective  action  will  be  taken.  Thread  is  an  indication 
of  what  action  is  expected  of  the  node  in  Next.  In  general  when  a  node 
receives  the  token  it  may  have  several  messages  ready  to  transmit,  Thread 
"suggests"  which  message  should  be  transmitted.  Section  5.3  explains  this 
in  more  detail. 

5.1.4     Flags. 

The  Flags  field  in  Table  5.1  contains  various  flags  and  codes  relating  to  the 
message.  The  Empty  Message  flag  indicates  that  the  message  contains  only 
the  header,  there  is  no  data  with  the  message.  The  Exception  flag  indicates 
that  the  message  is  an  exception  message.  The  Acknowledgement  Requested 
flag  indicates  that  the  receiving  node  should  reply  with  an  acknowledgement 
message.  The  Message  Type  is  a  code  for  the  type  of  message.  The  Exception 
Level  indicates  the  level  of  exception  processing.  Zero  is  the  normal  level 
so  a  message  with  level  0  is  a  normal  data  transfer  message.  This  is  much 
like  the  interrupt  level  found  in  most  micro-processors  (Section  5.4).  A 
message  with  level  0  and  the  Empty  Message  flag  set  is  considered  a  Null 
Message  and  contains  simply  the  header.  It  is  used  to  simply  pass  the 
token  on  primarily  by  the  Control  Node.  The  Overrun  type  is  used  by  a 
node  that  receives  the  token  when  it  should  have  a  message  to  transmit  but 
doesn't.  The  node  can  not  wait  for  the  message  to  be  made  ready  so  it 
transmits  an  empty  message  with  type  Overrun.  The  Exception  Poll  is  used 
to  poll  for  exception  messages  (see  Section  5.4).  Idle  Network  is  used  by  the 
control  node  to  indicate  that  it  expects  the  network  to  be  idle  following  this 
message.  Usually,  an  idle  network  indicates  that  the  token  has  been  lost 
and  recovery  procedures  should  be  initiated.  The  remaining  types.  Clear 
Test  and  Set,  Test  and  Set,  Test  and  Set  Reply  (0),  Test  and  Set  Reply  (1)), 
Positive  Acknowledgement,  and  Negative  Acknowledgement  are  used  for  low 
level  synchronization  between  the  nodes  see  Section  6.2.2. 

5.2      Access  control. 

On  a  GANGLIA  network  access  to  the  network  is  tightly  controlled.  The  com- 
munication (or  central)  node  (node  C  in  Figure  5.1)  is  primarily  responsible 
for  distributing  access.  In  a  larger  sense  the  communication  node  is  respon- 
sible for  the  integrity  of  the  network.  It  is  responsible  detecting  that  the 
token  is  lost  and  then  initiating  recovery.    It  is  an  important  part  of  the 


5.2.    Access  control. 21 

priority  scheme  that  filters  out  less  important  messages  during  the  systems 
response  to  exceptional  conditions.  And  it  is  responsible  for  switching  from 
one  scheduling  plan  to  another  as  the  systems  demands  change. 

5.2.1      Servo  cycles. 

Servo  loops  is  one  of  the  dominant  characteristics  of  robot  control  program- 
ming. Likewise  servo  loop  communication  plays  a  dominant  role  on  the 
network.  Under  normal  operation,  communication  on  the  network  is  di- 
vided into  communication  cycles.  Communication  cycles  occur  at  a  rate 
which  is  a  common  multiple  of  the  various  servo  loop  rates.  During  any 
one  cycle  some  set  of  the  nodes  will  have  servo  messages  (see  Section  3)  to 
transmit  and  some  nodes  will  have  steady  state,  changing  state  messages,  or 
exception  messages  to  transmit.  The  central  node  knows  to  a  large  extent 
what  messages  are  expected  during  any  cycle.  In  particular  it  knows  what 
servo  messages  will  be  transmitted  during  a  particular  cycle.  It  may  know 
what  steady  state  messages  will  be  transmitted  in  any  particular  cycle.  This 
scheduling  information  is  generated  when  the  system  is  compiled  and  placed 
in  tables  in  the  central  node. 

A  typical  communication  cycle  contains  the  following  steps: 

1.  The  central  node  sends  out  begin  servo  message  which  marks  the  be- 
ginning of  a  communication  cycle  and  passes  the  token  to  the  first  of 
the  nodes  to  transmit  servo  a  message. 

2.  The  indicated  node  transmits  its  data  in  a  message  that  indicates 
the  second  node  to  participate,  which  in  turn  passes  the  token  to  the 
third  node  and  so  on.  Each  node  knows  which  node  is  to  follow  from 
the  information  generated  at  compile  time  for  the  central  node.  A 
sequence  of  nodes  transmitting  messages  without  intervention  by  the 
central  node  is  called  a  thread.  The  Thread  field  in  the  header  of  the 
messages  indicates  the  current  thread  and  indicates  to  a  node  receiving 
the  token  which  node  is  to  receive  the  token  next  by  indexing  a  table 
in  the  CMMU.  The  final  node  in  the  servo  thread  passes  the  token 
back  to  the  central  node.  It  is  possible  that  several  servo  threads  may 
be  used  in  a  particular  cycle.  When  a  thread  is  complete  the  central 
node  begins  the  next  thread  by  passing  the  token  in  a  null  message 
perhaps  to  the  first  node  in  the  thread. 

3.  Following  the  last  servo  thread  the  central  node  then  transmits  an 
exception  poll  message.   This  message  indicates  to  all  the  nodes  that 
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exceptions  can  be  raised.  After  transmitting  the  message  central  node 
waits  a  small  amount  of  time  for  any  exception  to  be  raised.  Nodes 
wishing  to  transmit  an  exception  message  access  the  network  during 
this  interval  in  a  random  manner  Clearly  the  hope  is  that  exceptions 
will  be  rare  and  that  no  more  than  one  will  be  raised  during  any  one 
exception  poll  period.  However,  the  possibility  exists  for  contention  for 
the  access  to  the  network  during  this  period.  Resolving  this  contention 
is  discussed  in  Section  5.4.  The  central  node  may  poll  for  exceptions 
several  times  during  a  communication  cycle.  When  an  exception  is 
raised  it  may  very  likely  disrupt  the  normal  communication  cycles 
just  as  an  interrupt  disrupts  the  normal  flow  of  program  execution  in 
a  processor.  The  actions  taken  by  the  network  aad  the  control  node 
depend  on  the  exception  raised. 

4.  After  taking  care  of  servo  messages  and  exceptions  the  network  is  free 
to  handle  other  traffic.  From  the  compiled  schedule  information  the 
central  node  may  know  of  particular  messages  or  threads  (sequences 
of  messages)  that  should  be  transmitted  at  this  time.  It  can  start  a 
thread  by  passing  the  token  to  the  first  node  in  the  thread.  In  general 
the  central  node  can  simply  poll  the  nodes  for  any  message  they  wish 
to  send.  Polling  aU  the  nodes  may  not  be  possible  during  a  single 
communication  cycle,  especially  if  the  polls  produce  large  messages. 
In  this  case  the  central  node  can  continue  the  polling  in  the  following 
cycle  using  some  "fair"  schedule. 

Another  responsibility  of  the  communication  node  is  to  watch  for  a  lost 
token.  The  ganglia  protocol  requires  that  a  node  receiving  the  token  trans- 
mit a  message  immediately.  Under  normal  circumstances  the  node  has  one 
or  more  messages  ready  to  transmit  and  it  simply  picks  one  (using  the 
Thread  as  a  guide)  and  transmits  it.  If  a  node  has  no  messages  to  transmit 
it  transmits,  a  null  message  passing  the  token  on  as  indicated  by  the  Thread 
field.  The  node  can  indicate  that  it  was  unable  to  transmit  the  desired 
message  by  transmitting  an  Overrun  message.  If  nothing  else  is  appropriate 
the  token  is  passed  to  the  central  node.  It  is  possible  that  a  message  is  not 
successfully  received  by  the  node  receiving  the  token  in  this  case  the  token 
is  lost.  With  this  protocol  the  loss  of  the  token  can  be  detected  easily,  since 
if  the  central  node  does  not  have  the  token  and  the  network  is  idle,  then 
the  token  is  lost.  What  action  is  taken  by  the  control  node  when  it  detects 
a  lost  token  is  in  general  a  very  hard  problem  since  the  control  node  lacks 
information  as  to  how  the  token  was  lost  and  the  actual  state  of  the  system. 
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A  typical  response  might  be  to  wait  until  the  scheduled  time  for  the  of  the 
next  communication  cycle  and  then  start  it  that  cycle. 


5.3      Threads. 

Any  but  the  simplest  system  will  have  multiple  servo  loops  running  at  dif- 
ferent rates  and  may  have  complex  traffic  patterns  for  the  steady  state  and 
changing  state  messages.  The  central  node  has  the  task  of  managing  this 
traffic. 

Consider  the  example  system  in  Figure  5.1.  A  communication  cycle 
might  be  started  with  the  central  node,  C,  passing  the  token  to  A'  which 
transmits  its  message  and  passes  the  token  on  to  Y  which  does  the  same 
and  passes  the  token  to  Z  which  returns  the  token  to  the  central  node.  The 
central  node  polls  for  exceptions  and  then  polls  all  the  nodes  for  other  types 
of  messages  until  the  time  to  start  the  next  cycle.  This  sequence  of  passing 
the  token  from  C  to  A'  to  Y  to  Z  to  C  is  an  example  of  a  thread.  In  a  more 
complex  example  nodes  M  and  A'^  may  transmit  messages  in  a  servo  loop 
running  one  forth  as  often  as  the  above  loop.  Three  out  of  four  servo  cycles 
would  then  be  just  as  described  above  and  the  fourth  cycle  could  take  either 
of  two  forms,  a  single  thread  (C  to  X  to  F  to  Z  to  M  to  N  to  C)  or  two 
concatenated  threads  (C  to  A  to  y  to  Z  to  C  followed  by  C  to  M  to  iV  to 
C).  The  sets  of  nodes  in  the  various  threads  do  not,  of  course,  have  to  be 
distinct.  It  is  very  likely  that  some  nodes  will  be  part  of  several  threads  and 
on  each  of  the  threads  the  node  is  to  send  a  different  set  of  data.  This  leads 
to  some  ambiguity  for  a  node  which  receives  the  token,  what  information  is 
the  node  to  transmit?  That  is,  what  communication  thread  is  in  process. 
To  help  disambiguate  for  a  node  an  identifier  for  the  thread  is  included  as 
part  of  the  token,  this  is  the  Thread  field.  Thus  the  token  consists  of  an 
identifier  for  the  node  to  transmit  and  a  thread  identifier  which  essentially 
indicates  what  data  is  to  be  transmitted. 

Once  the  servo  part  of  a  communication  cycle  is  complete  the  central 
node  elicits  other  traffic  (steady  state  or  changing  state)  by  polling  the  nodes 
on  the  network.  In  a  simple  system  the  nodes  may  be  polled  repeatedly  until 
the  time  for  the  start  of  the  next  cycle.  In  general,  this  part  of  the  cycle  is 
not  so  simple.  There  may  be  too  many  nodes  to  poll  every  communication 
cycle.  There  may  be  priority  constraints  to  be  considered.  There  may  be 
traffic  patterns  that  can  or  should  be  used  to  advantage.  For  instance, 
during  the  change  of  state  of  the  system  it  may  be  known  that  certain 
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high  level  nodes  wiU  be  transmitting  large  amounts  of  data;  distributing 
new  parameter  tables,  stopping  unneeded  processes,  starting  new  processes, 
etc.  In  this  case  the  central  node  should  poll  these  high  level  nodes  more 
frequently.  There  may,  in  fact,  be  specific  sequences  for  which  threads  could 
be  established  to  speed  the  transition. 

Where  do  threads  come  from?  Threads  are  specified  by  the  system 
designer.  AH  scheduling  could  be  done  by  polling  but  by  turning  common 
sequences  into  threads  some  of  the  communication  bandwidth  is  saved.  An 
additional  benefit  is  that  the  thread  identifier  in  the  token  indicates  to  the 
receiving  node  what  sort  of  message  is  expected.  A  useful  tool  would  be  a 
GANGLIA  compiler  which  would  analyze  traffic  patterns,  set  up  threads,  and 
generate  schediding  tables  for  the  central  node. 


5.4     Exceptions. 

The  above  discussion  describes  normal  operation  and  ignores  exceptional 
conditions.  Exceptions  are  unscheduled,  often  undesirable,  events.  The 
problem  they  present  to  the  access  schediiler  is  that  since  their  occurrence 
cannot  be  predicted  they  play  havoc  with  any  schedule  the  central  node  has 
set  up.  One  solution  would  be  to  have  the  central  node  poll  the  individual 
nodes  for  exceptional  conditions.  One  hopes  that  the  events  are  uncommon 
so  that  the  repeated  polling  seems  wasteful  and  to  exacerbate  the  situation, 
exceptions  should  be  treated  with  a  high  priority  which  means  the  polling 
would  have  to  be  done  often.  Instead  a  special  message,  the  exception  poll, 
is  available.  Tliis  message  is  transmitted  by  the  central  node  when  it  is 
ready  to  receive  exceptions.  After  the  exception  poU  is  transmitted  the 
network  is  left  idle  for  a  short  interval  during  which  any  node  wishing  to 
raise  an  exception  begins  transmitting  the  exception  message.  This  places 
a  scheduling  constraint  on  the  central  node,  it  must  broadcast  an  exception 
poll  often  enough  to  satisfy  the  most  time  critical  exception  in  system. 

As  mentioned  previously,  contention  for  access  to  the  network  can  occur 
during  this  interval.  To  resolve  contention  a  jamming  preamble  is  affixed  to 
exception  messages.  The  highest  priority  exception  has  the  longest  pream- 
ble so  that  it  will  still  be  jamming  when  other  exception  nodes  have  begun 
transmitting  the  message  body.  While  transmitting,  nodes  listen  for  colli- 
sions and  stop  transmitting  when  any  collision  is  detected.  Thus  if  multiple 
exceptions  are  raised  simultaneously  then  all  but  the  highest  priority  node 
wiU  detect  a  collision  and  stop  transmitting  leaving  the  network  for  the 
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highest  priority  message.  This  protocol  has  the  curious  side  effect  that  the 
highest  priority  exception  has  the  longest  preamble  and  thus  the  longest  de- 
lay to  transmit.  The  delay  caused  by  the  jamming  preamble  is  small  though 
compared  to  the  communication  cycle  and  delays  caused  by  the  demands  for 
other  traffic.  Nonetheless,  this  contention  resolution  scheme  is  not  pleasing 
and  requires  more  attention. 

One  further  issue  needs  to  be  addressed,  what  is  to  be  done  when  the 
system  is  flooded  with  exceptions?  This  is  likely  when  truly  exceptional 
events  occur.  Consider  an  arm  that  unexpectedly  encounters  a  wall.  Force 
sensors  at  the  joints  and  end  effector  may  detect  excessive  forces  and  several 
nodes  may  try  to  raise  the  exception.  Meanwhile,  it  appears  to  position 
sensors  on  the  joints  that  the  actuators  are  failing  since  the  joints  do  not 
move  as  commanded,  more  exceptions  to  be  raised.  A  flood  of  exceptions 
like  this  is  likely  to  interfere  with  an  effective  response  to  the  real  crisis. 
Ganglia's  solution  is  to  adapt  the  technique  that  CPU's  use  to  handle  a 
flood  of  interrupts  -  prioritization  of  the  interrupts.  Each  exception  message 
has  a  priority  in  the  range  1  to  7  with  7  being  the  highest  priority.  Included 
as  part  of  the  exception  poll  is  the  current  exception  level  of  the  network. 
A  node  responds  to  the  exception  poll  only  if  it  has  a  pending  exception 
whose  priority  is  greater  than  the  networks  exception  level.  An  exception 
message  itself  includes  the  priority  of  that  exception  so  that  the  central 
node  can  raise  the  networks  level  appropriately.  Restoring  the  exception 
level  after  an  exception  has  been  properly  noted  or  handled  is  ultimately 
the  responsibility  of  the  central  node  but  the  node  raising  the  exception, 
other  affected  nodes  on  the  network,  and  higher  level  control  may  all  be 
involved  in  the  response  to  an  exception.  In  a  well  designed  system  the 
response  to  most  exceptions  will  be  planned  out  and  the  central  node  can 
restore  the  exception  level  when  the  plan  has  been  carried  out. 


Section  6 

The  CMMU 


The  Communication  Memory  Management  Unit  is  the  interface  between  a 
processor  in  a  node  and  the  GANGLIA  network.  The  CMMU's  collectively 
maintain  the  global  memory  and  individually  provide  access  to  the  network 
for  their  respective  processors.  A  CMMU  is  essentially  a  dual  ported  mem- 
ory management  unit.  It  has  three  major  functions: 

•  To  provide  access  to  the  local  portion  of  the  global  memory,  the  com- 
munication memory,  for  the  processor.  The  processor  reads  or  writes 
to  the  communication  memory  by  presenting  the  global  address  of  the 
desired  word  to  the  CMMU  which  translates  this  into  the  physical  lo- 
cation in  the  communication  memory  and  either  fetches  or  stores  the 
value  depending  on  the  operation. 

•  To  maintain  the  local  portion  of  the  global  memory  by  monitoring  the 
network  traffic  and  when  a  message  containing  a  data  item  which  is 
maintained  locally  is  received  the  data  is  stored  in  the  communication 
memory. 


• 


To  provide  access  to  the  network  for  the  processor.  If  the  processor 
needs  to  transmit  a  message  it  constructs  the  appropriate  data  item 
in  the  communication  memory  and  instructs  the  CMMU  to  transmit 
the  message.  When  the  CMMU  receives  a  token  appropriate  for  this 
message  the  message  is  transmitted.  In  this  regard  the  CMMU  is  also 
responsible  for  supporting  the  integrity  of  the  network  by  participat- 
ing in  the  ganglia  protocol.  When  the  token  is  passed  to  a  node  the 
CMMU  is  responsible  for  responding  immediately  with  the  appropri- 
ate message  or  a  nuU  message  if  nothing  is  appropriate.    When  the 
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processor  wishes  to  raise  an  exception  the  CMMU  is  responsible  for 
obeying  the  protocol  concerning  exceptions. 


6.1      Memory  map  table. 

The  core  of  the  CMMU  is  the  memory  m,ap  table  which  translates  global 
memory  references  to  communication  memory  locations  and  contains  other 
information  about  the  items  in  global  memory.  Table  6.1  shows  the  structure 
of  an  entry  in  the  map  table.  The  table  itself  is  maintained  primarily  by  the 
processor.  Programs  running  on  the  processor  know  what  data  items  are  of 
interest  and  which  they  will  want  to  transmit  and  they  must  prepare  the 
table  accordingly.  As  messages  are  received  and  transmitted  on  the  network 
the  CMMU  modifies  the  table  entries  accordingly. 

Table  is  accessed  by  associative  search  for  the  most  part.  The  keys  used 
for  the  search  consist  of  the  Name  and/or  Thread  and  various  of  the  Flags. 
The  entries  can  also  be  accessed  randomly  by  the  processor.  There  will  be 
times  when  more  than  one  of  the  entries  will  match  a  search  key.  In  this 
case,  the  chosen  entry  will  be  chosen  deterministically,  say  the  entry  lowest 
in  the  sequential  ordering  of  the  entries. 

The  fields  of  the  table  entries  are  described  in  more  detail  below. 

Name  is  the  global  name  of  the  data  item.  This  is  assigned  when  the 
system  is  compiled  and  the  same  name  is  used  throughout  the  network.  It 
is  possible,  in  fact  likely,  that  there  wiU  be  more  than  one  entry  in  the  table 
for  a  given  item,  as  will  be  seen  below. 

Thread  is  thread  identifier.  When  a  node  receives  the  token  the  CMMU 
searches  the  map  table  for  an  item  that  is  ready  to  transmit  and  has  Thread 
matching  the  thread  identifier  from  the  message  just  received.  One  of  the 
matching  items  is  then  transmitted. 

The  Flags  contciins  various  flags  associated  with  this  table  entry.  Ta- 
ble 6.2  describes  the  flags. 

Next/Source  holds  a  node  identifier.  If  this  item  is  being  transmitted 
this  is  used  for  the  Next  field  of  the  message.  If  this  item  is  received  the 
Source  field  from  the  message  is  stored  here. 

Cycle  is  the  cycle  time  stamp  for  the  field.  On  transmission  this  field  is 
included  in  the  message  and  on  reception  the  Cycle  field  from  the  message 
is  stored  here. 

Mthread  contains  the  Thread  field  for  the  message.  As  with  the  Cycle 
field  it  is  either  a  source  or  destination  depending  on  how  the  entry  is  used. 


28 


The  CMMU 


Field  Name 

Bytes 

Description 

Name 

3 

The  global  name  of  the  data  item. 

Thread 

1 

The  thread  identifier.  See  Section  5.3. 

Flags 

2 

Flags: 

Locally  Mapped 

Exception  Message 

Receive/ Transmit 

Pending 

Busy 

Done 

Interrupt  When  Done 

Disable  When  Done 

Map  W^en  Done 

Test  and  Set 

Next/Source 

2 

A  node  identifier  used  either  as  the  Next  field 
when  transmitting  or  to  hold  the  Source  field 
when  the  item  is  received. 

Cycle 

2 

A  time  stamp  from  the  last  time  the  item  was 
received. 

Mthread 

1 

The  Thread  Held  from  the  last  message  receive  for 
this  entry  or  to  be  transmitted  with  this  message. 

Mflags 

1 

The  Flags  field  from  the  last  time  the  item  was 
received  or  to  be  transmitted  with  the  message. 

Location 

4 

The  location  of  this  item  in  the  communication 
memory. 

Table  6.1:  Memory  Map  Table  Entry 
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Flag 

Description 

Locally  Mapped 

When  set  this  flag  indicates  that  the  entry  can 
be  used  in  memory  references  by  the  processor. 

Exception  Message 

When  set  this  flag  indicates  this  is  for  a  pend- 
ing exception  message.  This  message  may  be 
transmitted  when  the  CMMU  receives  an  ex- 
ception poll. 

Receive/Transmit 

This  flag  indicates  whether  this  entry  is  for 
receiving  or  transmitting  the  data  item. 

Pending 

When  set  this  flag  indicates  to  the  CMMU 
that  the  reception  or  transmission  of  this  item 
is  pending. 

Busy 

When  set  this  flag  indicates  that  the  CMMU 
is  using  the  table  entry.  Usually  this  means 
that  transmission  or  reception  is  in  process. 

Done 

This  flag  is  set  when  a  transfer  is  complete.  It 
is  cleared  when  the  Pending  flag  is  set. 

Interrupt  When  Done 

When  set  this  flag  indicates  that  the  CMMU 
should  interrupt  the  processor  when  the  Done 
flag  is  set. 

Disable  When  Done 

When  set  this  flag  indicates  to  the  CMMU 
that  the  pending  flag  should  be  cleared  when 
a  transfer  is  complete.  Thus  further  trans- 
fers via  this  table  entry  are  prevented  until  the 
Pending  flag  is  explicitly  set  by  the  processor. 

Map  When  Done 

When  set  this  flag  indicates  to  the  CMMU 
that  the  Locally  Mapped  flag  should  be  set 
when  a  transfer  is  complete. 

Table  6.2:  Memory  Map  Entry  Flags 
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M flags  contains  the  Flags  field  for  the  message.  Again  it  is  either  a  source 
or  destination  depending  on  how  this  entry  is  used. 

Location  is  the  address  in  the  communication  memory  where  this  item 
is  stored.  When  a  message  is  received  for  the  data  item  associated  with  this 
entry  the  data  portion  of  the  message  is  stored  starting  at  this  location  in 
the  communication  memory.  When  a  reference  from  the  processor  to  global 
memory  uses  this  entry  the  lowest  byte  of  the  virtual  address  is  added  to 
Location  to  obtain  the  actual  address  in  the  communication.  Note  that  the 
page  size  in  the  global  memory  is  256  bytes  but  the  Location  pointer  does 
not  have  to  be  on  a  page  boundary.  Depending  on  communication  memory 
design  considerations  it  may  have  to  be  aligned  on  a  word  boundary.  The 
reason  that  the  entries  in  the  map  table  do  not  correspond  to  memory  frames 
as  with  typical  memory  management  units  is  that  each  named  data  item 
(i.e.  the  data  transmitted  in  a  named  message)  is  one  page  in  the  global 
memory  but  most  of  the  items  are  much  less  that  than  256  bytes  and  the 
communication  memory  on  the  nodes  would  be  terribly  fragmented  if  each 
item  used  one  frame  of  the  memory. 

6.2      CMMU  operation. 

The  programs  on  the  processor  are  responsible  for  maintaining  the  memory 
map  table.  Typically,  a  program  (or  the  operating  system  on  the  node)  will 
set  up  the  map  table  entries  during  program  initialization  and  manipulation 
of  the  table  while  the  program  is  running  will  be  minimal,  although  dynamic 
changing  of  the  table  is  possible  and  even  desirable  in  some  cases.  The 
remaining  parts  of  this  section  describe  how  typical  functions  are  performed. 

6.2.1      Processor  references. 

For  the  processor  to  reference  a  global  data  item  in  the  communication 
memory  four  conditions  must  be  met.  There  must  be  a  map  table  entry 
with  its  Name  field  set  to  the  global  name  of  the  item.  This  is  true  if  the 
first  three  bytes  of  the  global  address  match  the  Name  field.  Second  the 
Location  field  of  the  same  entry  must  point  to  the  appropriate  location  in 
the  communication  memory.  The  Locally  Mapped  flag  must  be  set.  And  the 
Busy  flag  must  be  off,  i.e.  the  entry  must  not  be  in  use  by  the  CMMU.  K  these 
conditions  are  met  the  low  order  byte  of  the  global  address  is  added  to  the 
Location  field  and  the  resulting  address  is  used  to  access  the  communication 
memory.   The  Name  field  and  the  Location  field  must  be  explicitly  set  by 
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the  processor,  the  Locally  Mapped  flag  can  be  set  either  by  the  processor  or 
the  CMMU,  and  the  Busy  is  set  by  the  CMMU  when  it  is  using  the  entry. 
If  the  conditions  are  not  satisfied  a  page  fault  is  generated  to  the  processor. 
The  usual  response  if  the  entry  is  busy  is  for  ths  processor  is  to  postpone 
the  reference  until  the  entry  is  no  longer  busy.  This  is  rather  costly  to  the 
program  making  the  reference  and  if  it  is  on  a  critical  time  path  when  the 
fault  occurs  the  result  can  be  disastrous.  The  following  paragraphs  discuss 
ways  to  avoid  access  conflicts. 

6.2.2      Synchronization. 

There  are  basically  two  ways  to  avoid  simultaneous  access  to  a  data  item. 
The  first  is  to  synchronize  the  data  producing  process  and  the  data  con- 
suming process.  Many  of  the  activities  in  a  robot  control  system  are  syn- 
chronized around  various  control  cycles.  So  often  proaucing  and  consuming 
processes  are  often  already  synchronized.  For  example,  a  system  design  can 
be  such  that  a  particular  data  item  is  always  produced  and  transmitted  near 
the  end  of  a  cycle  allowing  the  consuming  processes  to  use  the  item  during 
the  beginning  of  a  cycle  without  fear  of  simultaneous  access.  The  second 
way  to  synchronize  is  to  let  the  producer  process  "drive"  the  consumer  pro- 
cess. For  instance  the  consuming  node  can  set  the  Interrupt  When  Done 
flag  which  will  cause  the  processor  to  be  interrupted  when  the  message  is 
received.  Assuming  it  does  not  need  the  data  for  "too  long"  it  can  use  the 
data  without  fear  of  interference  from  the  network.  If  the  two  processes 
cannot  be  easily  synchronized  it  is  possible  to  double-buffer  the  messages 
for  the  data  item.  Briefly,  this  is  accomplished  by  keeping  two  entries  for 
the  data  item,  one  is  mapped  into  the  local  address  space  and  is  available 
to  the  processor  the  other  is  marked  as  pending  and  is  ready  to  receive  an 
incoming  message.  The  local  program  is  responsible  for  manipulating  the 
table  entries  appropriately.  A  process  that  transmits  data  has  a  similar 
problem,  it  arises  because  there  is  some  time  between  when  a  data  item  is 
marked  as  waiting  for  transmission  (i.e.  its  Pending  flag  is  set)  and  when 
the  transmission  is  complete  and  the  area  of  communication  memory  can  be 
modified.  There  are  similar  solutions  also,  transmission  can  be  synchronized 
with  production  oflf  the  message  either  by  design,  or  by  using  interrupts  or 
by  double-buffering. 

The  purpose  of  the  Disable  When  Done  and  the  Map  When  Done  flags 
are  to  reduce  the  communication  overhead  to  the  local  program.  If  the 
producing  and  consuming  processes  are  synchronized  in  such  a  fashion  that 
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there  is  never  a  conflict  over  access  to  the  communication  memory  location 
for  the  a  data  item  then  a  single  entry  in  the  table  can  be  used  for  the  item 
and  the  Pending  and  Locally  Mapped  flags  can  be  left  on  at  all  times  (i.e.  the 
Disable  When  Done  and  Map  When  Done  flags  are  off).  In  this  case  the 
local  program  uses  the  communication  memory  area  according  to  its  schedule 
and  the  CMMU  will  transmit  or  receive  as  the  network  indicates  (i.e.  when 
the  message  or  token  arrives)  and  no  special  communication  overhead  is 
required.  If  however,  there  is  a  danger  of  the  CMMU  and  the  processor 
interfering  with  each  other  then  these  flags  can  help  to  avoid  conflict.  If  the 
Disable  When  Done  flag  is  on  then  when  a  message  is  received  or  transmitted 
the  Pending  flag  is  automatically  turned  off  upon  completion  so  that  the 
same  entry  (and  area  in  communication  memory)  can  not  be  used  again  by 
the  CMMU  until  explicitly  enabled  by  the  processor.  And  the  Map  When 
Done  flag  controls  access  by  the  processor.  If  it  is  on  then  the  Locally  Mapped 
flag  is  automatically  set  at  the  end  of  transmission  or  reception  so  that  the 
processor  will  be  able  to  access  the  data  as  soon  as  it  is  available  but  will 
fail  if  it  tries  to  access  it  prematurely. 

6.2.3     Transmitting  messages. 

To  transmit  a  data  item  the  processor  prepares  the  data  in  the  communi- 
cation memory  area  indicated  by  the  the  Location  field  of  the  entry.  If  the 
entry  is  mapped  into  the  local  space  then  this  is  accomplished  simply  by 
referring  to  the  locations  by  their  global  names  as  in  the  previous  section. 
Various  fields  in  the  map  table  entry  must  also  be  set.  The  Mthread  a.nd  Next 
fields  must  be  set  to  the  appropriate  values,  these  are  usually  determined 
when  the  system  is  compiled.  The  message  will  be  transmitted  when  a  token 
is  received  for  this  node  with  the  indicated  thread,  the  token  will  be  passed 
to  the  node  indicated  by  Next.  The  Cycle  field  should  be  incremented  to 
distinguish  this  transmission  of  the  data  from  previous  transmissions.  The 
flags  for  the  outgoing  message  should  be  set  appropriately  in  Mflags.  Fi- 
nally, the  Flags  m  the  table  entry  must  be  set.  In  particular.  Locally  Mapped 
should  probably  be  turned  off.  Receive / Transmit  shoxAA  be  set  to  transmit. 
Interrupt  When  Done  and  Disable  When  Done  should  be  turned  on  or  off, 
as  desired.  And  the  Pending  flag  should  be  turned  on. 

When  the  token  is  passed  to  this  node  with  the  indicated  Thread  field 
of  the  message  matching  the  Thread  field  in  the  table  entry  the  data  item 
will  be  transmitted.  During  transmission,  the  Busy  flag  will  be  turned  on. 
At  the  end  of  transmission  the  Busy  flag  will  be  turned  off  and  the  local 
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processor  will  be  interrupted  if  the  Interrupt  When  Done  flag  is  on  and  the 
Pending  flag  will  be  turned  off  if  the  Disable  When  Done  flag  is  on. 

As  mentioned  above  it  is  sometimes  necessary  to  double-buffer  data 
items.  When  a  node  responsible  for  generating  a  data  item  must  double 
buffer  it  sets  up  two  entries  in  the  map  table  for  the  same  global  data  item. 
The  Name,  Thread,  Mthread,  Mflags,  Next/Source,  and  many  of  the  Flags 
fields  would  be  the  same  in  the  two  entries.  The  Location  pointers  would 
point  to  distinct  areas  of  the  communication  memory.  Initially,  one  of  the 
entries  would  be  locally  mapped  and  the  other  is  neither  mapped  nor  marked 
as  pending.  The  processor  prepares  the  first  message  for  transmission  in  us- 
ing the  mapped  entry,  when  ready  the  entry  is  prepared  for  transmission 
and  the  Pending  flag  is  turned  on.  Now,  while  waiting  for  i;he  message  to  be 
transmitted  the  processor  can  prepare  the  next  occurrence  of  this  message 
by  turning  on  the  Locally  Mapped  flag  in  the  second  entry.  When  it  is  ready 
for  transmission  it  can  be  marked  as  pending  if  the  previous  message  has 
already  been  transmitted.  The  processor  can  then  map  the  first  entry  again 
and  prepare  the  next  message  using  this  entry.  To  keep  the  Cycle  field  in 
sequence  it  should  be  incremented  by  two  each  time  an  entry  is  used  for 
transmission.  This  scheme  can  be  extended  to  more  than  two  buffers  by 
using  more  entries  as  long  as  only  one  entry  is  mapped  at  any  time  and  only 
one  entry  is  Wciiting  for  transmission  at  any  time. 

6.2.4      Receiving  messages. 

To  receive  a  data  item  the  node  allocates  a  table  entry  and  an  area  in  the 
communication  memory  for  the  data  item,  points  the  Location  field  of  the 
entry  to  the  memory  area,  sets  the  Name  field  of  the  entry,  and  turns  on 
the  Pending  flag  and  clears  the  Receive/Transmit  flag  (to  indicate  receive). 
When  a  message  is  received  the  CMMU  searches  the  map  table  for  an  entry 
with  its  Pending  flag  set  and  its  Name  field  matching  the  Name  field  of  the 
incoming  message.  If  an  entry  is  found  then  the  Location  pointer  of  the  entry 
points  to  the  area  to  store  the  message.  If  the  Interrupt  When  Done  flag  is 
turned  on  then  the  processor  is  interrupted  when  the  message  is  successfully 
received.  The  Disable  When  Done  flag,  if  set,  causes  the  CMMU  to  turn  off 
the  Pending  flag  after  a  message  is  received.  This  prevents  another  incoming 
message  from  using  the  table  entry  until  the  processor  explicitly  enables  it 
again.  If  the  Map  When  Done  flag  is  on  the  CMMU  sets  the  Locally  Mapped 
flag  after  the  message  is  received.  Thus  the  data  item  will  automatically 
appear  in  the  processors  address  space.  The  other  fields  in  the  table  entry 
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are  set  according  to  the  incoming  message  so  the  processor  can  examine  the 
Mthread,  Source,  Cycle  and  message  flags  (Mflags)  if  desired. 

Double  buffering  for  received  data  items  is  similar  to  transmitted  items. 
Two  table  entries  are  set  up  with  the  same  Name  and  distince  Location  fields. 
One  of  the  entries  is  marked  as  pending  while  the  other  is  mapped.  The 
processor  uses  the  mapped  entry  while  the  CMMU  uses  the  pending  entry 
to  receive  an  incoming  message.  When  a  message  is  received  the  processor 
can  switch  the  two  entries,  as  desired.  Again  more  than  two  entries  can  be 
used  as  long  as  only  one  is  pending  and  only  one  is  mapped  at  any  time. 

6.2.5      Global  memory  test  and  set. 

The  most  common  primative  operation  for  inter-processor  synchronization 
is  the  test  and  set  operation.  Modern  micro-processors  include  test  and 
set  operations  in  some  form  [MoTo85]  [Inte86].  Micro-processor  busses 
facilitate  test  and  set  operations  by  allowing  a  bus  master  to  lock  the  bus, 
preventing  access  by  other  bus  masters,  for  a  series  of  operations  [Micr85] 
[Inte84].  The  important  condition  for  a  test  and  set  operation  is  that  the 
operation  be  atomic  thus  preventing  race  conditions  among  the  competing 
processors. 

It  is  important  that  ganglia  provide  a  test  and  set  facility.  To  insure 
the  integrity  of  a  test  and  set  operation  it  is  clear  that  for  any  particular 
synchronization  variable,  a  single  node  must  be  responsible  for  maintaining 
it.  Which  node,  is  not  so  important  but  a  single  node  must  maintain  the 
accurate  value  of  the  variable.  Other  nodes  must  "request"  a  test  and  set 
operation  on  the  variable  by  the  responsible  node  and  the  responsible  node 
must  perform  the  operation  as  an  atomic  operation  and  report  the  results. 
A  simple  approach  (from  the  standpoint  of  GANGLIA  design)  would  be  to 
have  the  processor  in  a  node  handle  the  request.  A  node  requiring  access 
to  a  resource  would  send  a  message  to  the  node  responsible  for  the  syn- 
chronization variable  for  the  resource  requesting  access  (and  simultaneously 
excluding  other  access).  The  processor  in  the  responsible  node  would  be 
interrupted  when  the  message  arrived  and  it  would  perform  the  necessary 
operations  including  setting  up  the  reply  message  for  transmission.  The  re- 
ply message  would  be  transmitted  at  some  point  when  the  token  was  passed 
to  the  responsible  node  and  the  requesting  node  would  then  get  its  reply. 
This  involves  a  great  deal  of  overhead,  the  responsible  node  is  interrupted, 
it  must  interpret  an  incoming  message,  perform  the  operation  and  prepare 
an  outgoing  message.  Meanwhile  the  requesting  node  must  wait  for  an  un- 
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determined  (possibly  long)  duration  for  the  reply.  A  preferable  approach 
for  implementing  the  test  and  set  is  to  have  the  CMMU  in  the  responsible 
node  handle  the  operation.  In  this  case  the  requesting  node  would  again 
send  a  test  and  set  request  to  the  responsible  node  and  the  token  would  be 
passed  to  the  responsible  node  as  part  of  the  request.  The  CMMU  receiving 
the  request  (along  with  the  token)  would  perform  the  test  and  set  operation 
and  report  the  results  immediately.  This  is  more  appealing  than  the  pre- 
vious approach  since  the  processor  in  the  responsible  node  is  not  involved 
in  the  operation  and  the  reply  is  returned  as  soon  as  possible  (i.e.  the  next 
message).  One  complication  is  that  the  CMMU  must  ensure  that  its  local 
processor  does  not  access  the  synchronization  variable  in  the  midst  of  the 
test  and  set  operation.  But  since  the  CMMU  controls  access  to  the  com- 
munication memory  by  the  local  processor  it  can  ensure  the  integrity  of  the 
operation. 

When  the  CMMU  receives  a  Test  and  Set  message  it  checks  the  map 
table  for  the  entry  indicated  by  the  Name  field  of  the  message.  If  none  is 
found  a  Negative  Acknowledgement  message  is  sent  and  the  token  is  passed 
to  the  originating  node.  If  an  entry  is  found  the  CMMU  checks  the  status 
of  the  Test  and  Set  flag  in  the  entry  and  sends  the  appropriate  response 
and  passes  the  token  to  the  originating  node.  The  Test  and  Set  flag  is  also 
turned  on.  The  Test  and  Set  flag  is  cleared  by  a  Clear  Test  and  Set  message. 

Placing  responsibility  for  the  test  and  set  on  the  CMMU  complicates 
its  design.  Up  to  this  point  the  network  functions  of  the  CMMU,  receiving 
and  storing  incoming  data  and  watching  for  the  token  and  transmitting 
messages,  were  quite  independent.  One  could  imagine  their  implementation 
as  independent  state  machines  within  the  CMMU.  However,  the  introduction 
of  the  test  and  set  operation  requires  that  the  receiving  and  transmitting 
portions  of  the  CMMU  must  be  intertwined. 

A  final  note  should  be  made  here.  Implementing  an  efficient  test  and  set 
operation  is  important  and  will  increase  the  flexibility  of  GANGLIA  systems. 
But  a  major  tenet  of  the  philosophy  behind  ganglia  is  that  robot  control 
systems  are  highly  synchronized  and  that  this  synchronization  can  be  used 
to  advantage  in  the  overall  system  design.  In  particular,  many  conflicts  for 
access  to  shared  resources  can  be  avoided  by  the  inherent  synchrony  of  the 
system  and  that  mechanisms  such  as  the  test  and  set  operation  will  not  be 
necessary  in  these  cases.  This  is  not  to  say  that  a  GANGLIA  system  does 
not  need  synchronization  primatives,  any  non-trivial  system  will  need  them, 
but  that  the  most  frequent  and  time-criticaJ  instances  of  potential  resource 
conflict  can  be  avoided  by  taking  advantage  of  the  synchrony  of  the  overall 
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system. 

6.2.6      Message  acknowledgements. 

Three  inter-related  assumptions  have  been  made  to  support  the  claim  that 
GANGLIA  is  a  feasible  architecture  for  robot  control  systems.  First,  a  com- 
munication medium  can  be  found  that  will  operate  with  suitable  reliability 
in  the  environment  of  a  robot.  It  need  not  be  totally  reliable  but  it  must  be 
sufficient  to  support  the  system.  Second,  much  (if  not  the  majority)  of  the 
communication  traffic  is  such  that  an  occasional  lost  packet  can  be  toler- 
ated. This  is  reasonable,  since  much  of  the  data  would  be  smoothly  varying 
and  would  be  continuously  reported.  Third,  distributed  programs  can  be 
developed  to  effectively  operate  in  this  environment.  It  is  unreeisonable  to 
assume  that  these  three  assumptions  hold  at  all  times.  Some  data  must 
be  successfully  transmitted,  the  code  for  down-loaded  program  for  example. 
Sometimes  a  node  must  know  "beyond  a  reasonable  doubt"  that  another 
node  has  received  and  accepted  some  data.  Ganglia  can  support  normal 
data-gram  messages  (see  Section  5.1)  so  handshaking  protocols  available 
on  regular  networks  are  available  on  GANGLIA.  The  CMMU  further  assists 
reliable  communication  in  the  following  ways. 

The  CMMU  keeps  a  copy  of  the  last  successfully  received  or  transmitted 
message  header  regardless  of  whether  or  not  the  data  in  the  message  was  of 
interest  to  the  node.  In  addition,  the  CMMU  can  interrupt  the  processor  if 
the  network  is  ever  idle.  The  ganglia  protocol  specifies  that  the  network  is 
never  idle  except  at  times  specified  by  the  control  node.  At  other  times  the 
network  will  only  go  idle  if  the  node  indicated  in  the  token  fails  to  receive 
the  token.  Thus  if  node  X,  in  Figure  5.1,  wishes  some  assurance  that  a 
message  sent  to  M  is  received  it  can  pass  the  token  to  M  along  with  the 
message  and  then  monitor  the  network  using  these  features  of  the  CMMU. 
If  the  network  goes  idle  and  the  last  header  is  the  message  to  M  then  it  is 
likely  that  the  message  was  not  received  by  M.  K  it  does  not  go  idle  or  it 
goes  idle  after  the  message  is  transmitted  but  other  traffic  was  observed  then 
the  message  was  received  by  M.  This  scheme  can  be  extended  to  sequences 
of  messages.  A  node  can  request  to  be  informed  if  the  network  goes  idle 
while  the  sequence  of  messages  is  in  progress.  If  the  network  becomes  idle 
then  the  last  observed  header  may  indicate  where  the  sequence  failed.  If 
the  network  never  goes  idle  during  the  sequence  then  all  the  messages  were 
successfully  transmitted.  Note  that  the  fact  that  the  node  indicated  by  the 
token  successfully  transmits  indicates  that  it  received  the  message  but  does 
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not  indicate  that  any  other  node  necessarily  received  the  token.  Thus  the 
above  scenarios  give  no  assurance  that  the  messages  were  received  by  all 
interested  parties.  Also  note  that  a  node  responding  to  the  token  indicates 
only  that  the  message  was  not  damaged  during  the  transmission.  It  does 
indicate  that  the  receiving  node  agrees  in  any  sense  with  the  message  or 
understands  it. 

In  a  manner  similar  to  the  test  and  set  operation  described  above  a 
node  can  request  that  the  CMMU  of  the  noded  receiving  the  token  respond 
immediately  with  an  acknowledgement.  This  is  a  more  familiar  and  direct 
form  of  acknowledgement  which  can  be  used  if  desired.  When  a  node  receives 
a  message  with  its  Acknowledgement  Requested  Q^aL,g  set  the  CMMU  replys  to 
the  originating  node  with  an  acknowledgement.  If  there  is  an  entry  matching 
the  data  item  in  the  Name  field  of  the  message  then  the  CMMU  replies  with 
a  Positive  Acknowledgement  message  otherwise  it  replies  with  a  Negative 
Acknowledgement  message. 


Section  7 


Alternatives 


This  section  presents  some  alternatives  to  parts  of  the  GANGLIA  protocol 
that  are  interesting  but  not  included  in  the  current  design. 


7.1      Multiple  messages  per  token. 

The  IEEE  standard  (IEEE  802.4)  for  token  bus  [Stal87]  does  not  include 
the  token  as  part  of  each  message.  Instead,  when  a  node  receives  the  token 
it  transmits  as  many  messages  as  it  can  and  then  passes  the  token  on. 
There  is  a  time  limit  on  how  long  a  node  can  hold  the  token  to  provide 
some  "fairness"  in  access  to  the  network.  A  similar  approach  could  be  taken 
within  GANGLIA.  It  would  reduce  the  overhead  on  each  message  by  removing 
the  token.  It  could  also  ease  the  burden  on  the  control  node  of  knowing  in 
detail  the  communication  requirements  of  cJl  the  nodes.  The  control  node's 
task  would  become  that  of  providing  adequate  and  timely  access  to  the 
various  nodes,  the  individual  nodes  would  take  on  more  of  the  responsibility 
of  what  to  transmit.  On  the  other  hand,  the  actions  taken  by  the  CMMU 
upon  receiving  the  token  are  more  ambiguous,  thus  creating  uncertainty  in 
the  schedule  of  access  to  the  network  and  in  general  reducing  the  central 
node's  control  over  the  network.  This  alternative  presents  an  interesting 
area  for  investigating  the  trade-off  of  deterministic  control  vs.  flexibility  on 
the  network. 

Note  that  to  some  extent  multiple  messages  per  token  can  be  achieved 
in  the  current  design  of  ganglia.  A  node  may  pass  the  token  to  itself, 
nothing  in  the  protocol  prevents  this.  So,  a  node  can  prepare  a  sequence  of 
messages  to  be  transmitted  and  when  the  token  arrives  transmit  each  one 

38 


7.2.    General  requests. 39 

but  passing  the  token  back  to  itself.  On  the  finaJ  message  of  the  sequence 
the  node  passes  the  token  on  to  the  appropriate.  This  scheme  does  not  save 
any  overhead  on  each  of  the  messages. 

7.2  General  requests. 

An  alternative  that  represents  a  fundamental  change  in  the  nature  of  mes- 
sages in  GANGLIA  is  to  turn  the  messages  into  more  general  requests.  Cur- 
rently, the  typical  message  in  GANGLIA  is  a  request  to  store  the  data  con- 
tained in  the  message  in  the  appropriate  global  memory  location.  The  Test 
and  Set  and  Acknowledge  Request  messages  are  special  requests  to  the  re- 
ceiving node  which  require  an  immediate  response  from  the  CMMU  in  the 
node.  More  general  request  could  be  implemented,  for  example  a  request  to 
report  the  current  value  of  the  global  data  item  (essentially  a  global  memory 
read  operation)  or  a  request  to  compare  the  data  in  the  global  memory  with 
the  data  in  the  message  and  return  the  result  or  a  request  to  perform  logical 
operation  using  the  global  memory  item  and  the  data  in  the  message.  Such 
a  protocol  would  be  more  flexible  than  the  current  ganglia  protocol  but 
would  complicate  the  CMMU  substantially. 

7.3  Flow  Control. 

The  GANGLIA  protocol  addresses  only  slightly  the  problem  of  flow  control, 
that  is  reducing  the  network  traffic  when  the  network  is  heavily  loaded.  The 
reason  that  flow  control  is  given  little  consideration  is  because  the  access 
scheduling  is  deterministic  and  thus  nominally  always  under  control.  During 
exception  processing  and  when  the  system  is  changing  state  one  can  imagine 
that  the  amount  of  traffic  will  be  too  much  to  be  effectively  manage.  One 
response  to  this  already  in  the  protocol  is  Exception  Level  on  exception 
messages  which  filters  out  low  priority  exceptions  when  the  network  is  busy 
with  more  important  activities.  This  notion  can  be  extended  to  all  messages. 
Every  message  can  have  an  exception  level  associated  with  it  and  before  a 
node  transmits  a  message  it  compares  the  level  of  the  message  with  the 
current  level  of  the  network.  If  the  network  priority  is  too  high  the  CMMU 
simply  passes  on  the  token  without  the  data  thus  saving  some  of  the  networks 
bandwidth. 

A  more  general  approach  is  to  associate  a  time  interval  with  each  message 
and  if  the  time  interval  between  consecutive  times  that  a  node  receives 
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the  token  exceeds  this  given  interval  then  the  message  is  not  transmitted, 
although  the  token  is  passed  on.  Preliminary  investigations  [Dlxo87]  of  this 
technique  indicate  that  it  provides  upper  bounds  on  the  interval  between 
times  the  node  receives  the  token  -  a  desirable  property. 

Flow  control  is  of  considerable  interest  since  effective  flow  control  scheme 
can  reduce  the  detailed  knowledge  required  by  the  control  node  to  schedule 
access  to  the  network.  If  the  various  nodes  can  police  themselves  then  the 
control  node  can  be  more  generous  with  access  to  the  network. 


Section  8 

Conclusions 


This  report  describes  a  novel  architecture  for  robot  controllers.  The  success 
of  the  architecture  depends  on  hardware  feasibility,  the  ability  of  the  protocol 
to  control  the  network  traffic,  while  satifying  the  real-time  constraints  of 
the  system,  and  the  programmability  of  the  system.  Of  the  three  items  the 
protocol  is  presented  in  most  detail  in  this  report. 

Much  work  needs  to  be  done.  Current  efforts  are  focused  on  the  develop- 
ment of  a  system  for  control  of  the  Utah/MIT  hand  which  provides  as  a  side 
benefit  a  useful  tool  for  the  simulation  of  ganglia.  The  core  of  the  system 
is  structure  for  managing  periodically  updated  data.  Although  developed 
independently  for  the  hand's  control  system  these  periodic  queues  are  used 
very  much  as  the  global  memory  of  ganglia  would  be  used.  Thus,  this  work 
provides  experience  with  the  style  of  programming  envisioned  on  a  GANGLIA 
system  and  thus  provides  important  insight  into  the  viability  of  GANGLIA  as 
an  architecture.  In  addition,  statistics  on  the  use  of  these  queues  provide  a 
basis  for  analysis  of  the  traffic  patterns  in  a  ganglia  system. 

The  next  steps  are  to  simulate  ganglia  and  to  investigate  it  analytically. 
Longer  term  plans  call  for  research  into  the  hardware  feasibility  and  then 
the  implementation  of  a  GANGLIA  prototype. 
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